1. Field of the Invention
The present invention relates to an apparatus and a method for placing (or setting: hereinafter “place” includes “set”) and/or deleting a marker in/from a scene in a case of calculating a position and orientation of a shooting viewpoint by using information about image coordinates in a captured image of markers placed in the scene.
2. Description of the Related Art
In recent years, studies on mixed reality have been actively conducted for the purpose of smoothly combining a physical space and a virtual space. An image display apparatus displaying mixed reality is realized by a video see-through method. In this method, an image of a virtual space (e.g., virtual objects or character information rendered by computer graphics) generated in accordance with a position and orientation of a viewpoint of an image capturing unit, such as a video camera, is mixed on an image of a physical space captured by the image capturing unit, and the mixed image is displayed.
As applications of such an image display apparatus, new fields different from conventional virtual reality are expected: for example, mixing body conditions on a body surface of a patient in order to aid an operation; and mixed reality games of fighting with a virtual enemy floating in a physical space.
Requirements common to these applications include how accurately the registration between a physical space and a virtual space should be performed, and considerable efforts have been made for this purpose. The problem of registration in mixed reality results in a problem of calculating a position and orientation of a shooting viewpoint in a scene (that is, in a world coordinate system).
In order to solve this problem, typically performed is obtaining a position and orientation of a shooting viewpoint in a scene by placing a plurality of markers in the scene and using coordinates in the world coordinate system of the markers and projection coordinates in an image captured by an image capturing unit.
The method of calculating a position and orientation of a shooting viewpoint based on a combination of coordinates in the world coordinate system of markers in a scene and projection coordinates thereof in a captured image has been traditionally suggested in a computer vision field and so on (e.g., see R. M. Haralick, C. Lee, K. Ottenberg, and M. Nolle, “Review and analysis of solutions of the three point perspective pose estimation problem,” International Journal of Computer Vision, vol. 13, no. 3, pp. 331-356, 1994 (hereinafter “Haralick et al.”); and D. G. Lowe: “Fitting parameterized three-dimensional models to images,” IEEE Transactions on PAMI, vol. 13, no. 5, pp. 441-450, 1991) (hereinafter “Lowe”).
Uchiyama, Yamamoto, and Tamura, “A Robust Registration Method for Merging Real and Virtual Worlds—Combining 6 DOF Sensor and IPC Algorithm—,” TVRSJ, Vol. 1, pp. 119-126, 2003 (hereinafter “Uchiyama et al.”) describes an example of applying such a method to registration in mixed reality. This document discloses a method of correcting, by using the method disclosed in Lowe, a position and orientation of a shooting viewpoint based on a position and orientation of a shooting viewpoint including an error obtained by measurement using a sensor so that coordinates in the world coordinate system of markers projected on a captured image match with image coordinates of the markers actually existing in the captured image.
One factor of determining registration accuracy in the above-described method (position matching between a physical space and a virtual space) is “a method of placing markers in a scene.”
For example, in the method described in Uchiyama et al., an error in a position and orientation of a shooting viewpoint cannot be corrected if no marker can be captured, so that registration accuracy decreases. Therefore, the registration accuracy depends on whether markers are placed in such a manner that one or more markers can always be captured from an estimated shooting position.
Also, in the method described in Uchiyama et al., when an “orientation” of a shooting viewpoint includes an error due to an error included in coordinates in the world coordinate system of a marker and projected coordinates of the marker, the registration accuracy in a space far from the shooting viewpoint relative to the marker is significantly lower than near the marker (see FIG. 3: a physical space three-dimensionally deviates from a virtual space significantly). On the other hand, when a “position” of a shooting viewpoint includes an error due to an error in coordinates in the world coordinate system of a marker and projected coordinates of the marker, the registration accuracy in a space close to the shooting viewpoint relative to the marker is significantly lower than near the marker (see FIG. 4: an image of a physical space two-dimensionally deviates from an image of a virtual space significantly). Therefore, whether a marker is placed near a space is a factor in determining the registration accuracy in the space.
In view of the above-described things, highly accurate registration can be effectively realized by evenly placing markers in a scene.
However, an image characteristic of a marker in a shooting range may not be adequately captured depending on a position and orientation of a shooting viewpoint. For example, if a marker is planer, an image characteristic of the marker cannot be adequately captured when the marker is shot from a direction orthogonal to a normal of the plane. Of course, the image characteristic of the marker cannot be captured from a position where the marker cannot be seen because it is hidden in an object. Therefore, when “evenly placing markers,” it is important to place the markers while considering the orientation of the markers so that a required number of markers whose image characteristic is adequately captured from an arbitrary position/orientation can be captured, in addition to evenly placing the markers in a scene.
However, if the markers are randomly placed or set in a scene, another disadvantage besides the registration accuracy occurs, for example, an increase in load of detecting markers and degradation in beauty of the scene. The latter problem should be avoided, that is, markers spoiling an appearance should be minimized in a physical space because mixed reality observed by a user includes the physical space as well as a virtual space. Therefore, markers should be “adequately” and evenly placed.
Accordingly, in a known method, markers are adequately and evenly placed in accordance with guidelines generated from some findings. For example, Japanese Patent Laid-Open No. 11-136706 (U.S. Pat. No. 6,522,312) discloses a method of adequately and evenly placing markers while decreasing a distribution density of markers in a space far from a shooting viewpoint and increasing a distribution density of markers near the shooting viewpoint.
However, the “guidelines generated from some findings” indicates only a desired tendency of placement of markers. Therefore, specific positions where markers should be placed are not indicated even if guidelines for placement of markers are given. That is, a method of placing markers determining registration accuracy has been left to the discretion of a person (user) who places markers.